Makeup simulation assistance apparatus, makeup simulation assistance method, and non-transitory computer-readable recording medium storing makeup simulation assistance program

ABSTRACT

A makeup simulation assistance apparatus includes: a simulation image generator that generates a simulation image obtained by superimposing, on a facial image picked up by an image pickup unit that picks up an image of a face of a user as the facial image, a makeup image showing a state of makeup as being applied to a facial component of the face; a simulation image output unit that outputs the generated simulation image; an interest degree sensor that senses an interest degree of the user for the makeup image, based on a facial image picked up by the image pickup unit when the simulation image is output; and an interest degree information output unit that outputs interest degree information indicative of the sensed interest degree.

BACKGROUND 1. Technical Field

The present disclosure relates to a makeup simulation assistanceapparatus, a makeup simulation assistance method, and a non-transitorycomputer-readable recording medium storing a makeup simulationassistance program.

2. Description of the Related Art

Manufacturers that manufacture and sell cosmetics as an exemplaryproduct relating to makeup often employ a selling model of developing anoriginal retail cosmetics shop in a department store or a supermarket(shop in shop), and selling products through conversations withcustomers, i.e., by so-called face-to-face selling. In the face-to-faceselling of cosmetics, for example, a beauty advisor (BA) of a retailcosmetics shop observes the skin or the like of a customer to givecounsel based on the observation result, and selects cosmetics assumedto be the most suitable for the customer.

Conventionally, at retail cosmetics shops, various tools for assistingin face-to-face selling of beauty advisors have been used. As anexemplary tool for assisting in face-to-face selling of a beautyadvisor, there is proposed a technique of simulating the customer's facewith makeup on a computer without actually applying makeup on thecustomer's face (for example, see PTL 1).

In the technique disclosed in PTL 1, a facial feature analysis and askin color analysis of the user's face are conducted based on apicked-up image of the user's face. Based on the result of the facialfeature analysis and that of the skin color analysis, information onproducts suitable for the facial feature and the skin color of theuser's face is displayed, and a simulated user's face with virtualmakeup using the products is displayed.

CITATION LIST Patent Literature

PTL 1: Unexamined Japanese Patent Publication No. 2011-22733

As the technique disclosed in PTL 1, various tools assisting inface-to-face selling are developed in order to ultimately lead customersto purchase of cosmetics. However, practically, use of such varioustools assisting in face-to-face selling unfortunately fails to lead thecustomer to purchase of cosmetics unless the beauty advisor identifieslatent needs of the customer for makeup and provides information thatthe customer wants by bringing out a customer's concern relating tomakeup. Practically, whether a beauty advisor can identify latent needsof a customer for makeup and efficiently lead the customer to purchaseof the cosmetics largely depends on the experience and skill of thebeauty advisor. Furthermore, in order to identify the latent needs ofthe customer for makeup, it may be possible for a salesclerk to ask acustomer questions. However, a salesclerk attending on a customer oftencannot afford the time for providing such a service. While it has beenconventionally proposed to actively simulate virtual makeup in sellingcosmetics, such technique only provides displaying the drawing result,and sensing the honest feeling of the customer depends on the skill ofthe salesclerk.

The above problems are not specific to the situation where a beautyadvisor (BA) is in face-to-face selling. In the distribution industry ofcosmetics, the proportion of shops in which a customer determines onhis/her own and purchases cosmetics and stay-at-home shopping via theInternet and the like (hereinafter referred to as no-salesclerk-selling)is increasing. Accordingly, it is assumed that there is an increasingdemand for an apparatus or a program not just performing simulation ofmakeup, but also assisting in identifying the latent needs of a customeror in making a decision.

SUMMARY

One non-limiting and exemplary embodiment provides a makeup simulationassistance apparatus, a makeup simulation assistance method, and anon-transitory computer-readable recording medium storing a makeupsimulation assistance program, each assisting in face-to-face sellingand no-salesclerk-selling in order to efficiently derive purchase of aproduct relating to makeup.

In one general aspect, the techniques disclosed here feature a makeupsimulation assistance apparatus including: a simulation image generatorthat generates a simulation image obtained by superimposing, on a facialimage picked up by an image pickup unit that picks up an image of a faceof a user as the facial image, a makeup image showing a state of makeupas being applied to a facial component of the face; a simulation imageoutput unit that outputs the simulation image being generated; aninterest degree sensor that senses an interest degree of the user forthe makeup image, based on a facial image picked up when the simulationimage is output; and an interest degree information output unit thatoutputs interest degree information indicative of the interest degreebeing sensed.

The present disclosure can assist in face-to-face selling andno-salesclerk-selling to efficiently derive purchase of a productrelating to makeup.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

Note that, these generic or specific aspects may be realized as asystem, an apparatus, a method, an integrated circuit, a computerprogram, or a recording medium, or as an arbitrary combination of asystem, an apparatus, a method, an integrated circuit, a computerprogram, and a recording medium.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a functional configuration of a makeupsimulation system in face-to-face selling according to a presentexemplary embodiment;

FIG. 2A is an exemplary display mode of a user's window in a makeupsimulation;

FIG. 2B is an exemplary display mode of a salesclerk's window in amakeup simulation;

FIG. 3 is a table showing interest information according to the presentexemplary embodiment;

FIG. 4 is a makeup candidate determination table according to thepresent exemplary embodiment;

FIG. 5A is a table showing user information according to the presentexemplary embodiment;

FIG. 5B is a table showing user information according to the presentexemplary embodiment;

FIG. 6 is a flowchart showing an operation of a salesclerk's sideterminal apparatus according to the present exemplary embodiment;

FIG. 7A is an exemplary display mode of the user's window at the startof face-to-face selling;

FIG. 7B is an exemplary display mode of the salesclerk's window at thestart of face-to-face selling; and

FIG. 8 is a block diagram showing a functional configuration of themakeup simulation system in no-salesclerk-selling according to thepresent exemplary embodiment.

DETAILED DESCRIPTION

In the following, with reference to the drawings, a detailed descriptionwill be given of a present exemplary embodiment. FIG. 1 is a blockdiagram showing a functional configuration of makeup simulation system10 according to an exemplary embodiment of the present disclosure.Makeup simulation system 10 is used, for example, as a tool forassisting in face-to-face selling of a beauty advisor (hereinafterreferred to as the “salesclerk”) at a retail cosmetics shop, that is,the selling manner that sells products through conversations with acustomer (corresponding to the “user” in the present disclosure,hereinafter referred to as the “user”).

Makeup simulation system 10 includes customer's side terminal apparatus100 (a tablet terminal) and salesclerk's side terminal apparatus 200 (agraphic tablet terminal) connected to each other through a communicationline. Customer's side terminal apparatus 100 is, for example, installedin portrait orientation opposing to the user, and displays a simulationimage of a face with makeup using the user's face. Salesclerk's sideterminal apparatus 200 assists in face-to-face selling to efficientlylead the customer to purchase of a product relating to makeup(cosmetics), when customer's side terminal apparatus 100 displays thesimulation image. Note that, customer's side terminal apparatus 100 isnot specified to a tablet terminal, and may be fixed to the counter inthe retail cosmetics shop.

Customer's side terminal apparatus 100 (corresponding to the “makeupsimulation assistance apparatus” of the present disclosure) includesimage pickup unit 110 and first display 120.

Note that, while not shown in the drawings, customer's side terminalapparatus 100 includes an illumination unit as illumination means, aflash unit as flash means, and a microphone unit as microphone means.Further, while not shown in the drawings, customer's side terminalapparatus 100 further includes, for example, a CPU (Central ProcessingUnit) as a processor, a storage medium such as ROM (Read Only Memory)storing any control program, work memory such as RAM (Random AccessMemory), and a communication circuit. In this case, the functions of theabove-described units are realized by the CPU executing the controlprogram.

Image pickup unit 110 picks up an image of the user's face as a facialimage. Image pickup unit 110 may be, for example, a CMOS (ComplementaryMetal-Oxide Semiconductor) camera, a CCD (Charge Coupled Device) cameraor the like. Image pickup unit 110 outputs the picked up facial image tosalesclerk's side terminal apparatus 200.

First display 120 is a liquid crystal display or an organic EL (organicelectroluminescence) display being great enough in size to display theuser's face at the actual scale, and displays a simulation image outputfrom customer's side terminal apparatus 100. FIG. 2A shows an exemplarydisplay mode of user's window containing simulation image 400. As shownin FIG. 2A, simulation image 400 includes facial image 410 picked up byimage pickup unit 110, and table image 420 with which, for the makeuptarget-site (e.g., the eyes, the mouth, or the like, hereinafterreferred to as the “facial component”), a color of a makeup item(lipstick in the example shown in FIG. 2A) can be selected. Here, 430 isa color image representing the color of the makeup item, the color beingcurrently selected by the user.

First display 120 accepts operations of the user or the salesclerk oncustomer's side terminal apparatus 100. To this end, first display 120is structured as a touchscreen monitor by a touchscreen being disposedto cover the window. Note that, the mechanism of accepting theoperations of the user or the salesclerk is not specified to thetouchscreen monitor, and a manipulation unit may be separately provided.

Salesclerk's side terminal apparatus 200 includes simulation imagegenerator 210, simulation image output unit 220, interest degree sensor230, interest degree information output unit 240, makeup candidateinformation output unit 250, second display 260, facial componentpositional information storage 270, facial position sensor 280, facialcomponent specifier 290, determination unit 300, interest degree historyinformation recorder 310, and interest degree history informationstorage 320.

Note that, while not shown in the drawings, salesclerk's side terminalapparatus 200 includes, for example, a CPU (Central Processing Unit) asa processor, a storage medium such as ROM (Read Only Memory) storing acontrol program (corresponding to the “makeup simulation assistanceprogram” of the present disclosure), work memory such as RAM (RandomAccess Memory), and a communication circuit. In this case, the functionsof the above-described units are realized by the CPU executing thecontrol program.

In response to any operation of the user or the salesclerk on customer'sside terminal apparatus 100, simulation image generator 210 generates asimulation image which is obtained by superimposing, on a facial imageoutput from image pickup unit 110 of customer's side terminal apparatus100, a makeup image showing the makeup state where makeup is applied onany facial component of the face contained in the facial image.

Simulation image output unit 220 outputs the simulation image generatedby simulation image generator 210 to first display 120 of customer'sside terminal apparatus 100.

Based on a facial image picked up by image pickup unit 110 when thesimulation image is output from simulation image output unit 220,interest degree sensor 230 senses a match degree between a facialcomponent in which the user is interested and a facial componentcorresponding to the makeup image contained in the simulation image,thereby sensing a user's interest degree for the makeup image.

Here, the match degree between the facial component in which the user isinterested and the facial component corresponding to the makeup imagecontained in the simulation image refers to the degree relating towhether or not they match with each other in the type of the facialcomponent (the eyes, the mouth, or the like). The facial componentcorresponding to the makeup image contained in the simulation imagerefers to a facial component strongly related to a makeup item(hereinafter referred to as the “makeup part”, for example, lipstick,eyeshadow, blusher, and the like) applied on the facial image. Forexample, for lipstick, the mouth (lips) is the facial component stronglyrelated, and for eyeshadow, the eyes are the facial components stronglyrelated. Further, in the case there exist a plurality of facialcomponents corresponding to the makeup image contained in the simulationimage, based on the temporal relationship between the time point atwhich the facial component is displayed on the simulation image or thetime point at which a change is made in the display, the matching degreeas to part of the facial components may be determined. Note that, thefacial components include eyes (right and left), the eyebrows (right andleft), the eyelashes (right and left), the mouth, the lips (upper andlower), the teeth, the nose, the cheeks (right and left), the ears(right and left), the contour, the temples, the hair and the like. Thefacial components may be classified in more detail, such as corners ofthe mouth, around the mouth, around the eyes, and are not specifiedthereto.

Specifically, interest degree sensor 230 senses the interest degree foreach facial component of the facial image contained in the simulationimage. For example, interest degree sensor 230 senses, by any knownmeans, a facial component positioned in the direction of the line ofsight of the user, and senses that the interest degree of the facialcomponent is greater as the accumulation time during which the directedline of sight stays is longer. FIG. 3 shows interest informationtemporarily generated by interest degree sensor 230 upon sensing thefacial component positioned in the direction of the line of sight of theuser. The interest information includes, as items, the date of visit ofthe user, the facial component, and the interest degree. The value inthe item “interest degree” is set to “high” when the accumulation timeduring which the directed line of sight of the user stays is longer thana predetermined threshold value or relatively longer. On the other hand,the value in the item “interest degree” is set to “low” when theaccumulation time during which the directed line of sight of the userstays is shorter than a predetermined threshold value or relativelyshorter. In the example shown in FIG. 3, while the user's interestdegree for the eyes is high, the user's interest degree for other facialcomponents such as the nose, the cheeks, the lips, and the like is low.That is, it can be estimated that the user is potentially interested inthe eyes while a makeup simulation of lipstick is being performed. Notethat, the interest degree may not be a binary determination such as“high” and “low”. In accordance with the length of the accumulationtime, higher continuous values or non-continuous values may be set whenthe time is longer.

Note that, interest degree sensor 230 may sense, by any known means, thefacial component of the face which is set to oppose to the image pickupunit 110 by the user, and may sense that the interest degree for thefacial component is higher as the accumulation time during which thefacial component is set opposite to image pickup unit 110 is longer.Further, interest degree sensor 230 may sense, by any known means, adeformation degree of a facial component caused by the user, and maysense that the interest degree for the facial component is higher as thedeformation degree is higher. For example, when a makeup simulation fora plurality of facial components (for example, the eyes and the mouth)is being performed, when the deformation degree for around the mouth ishigh, it can be estimated that the user is interested in a makeupsimulation of lipstick which is the makeup part for the mouth. Here, adeformation of a facial component may include: for the eyes,opening/closing the eyelids, shifting the line of sight, frowning andthe like; and for the mouth, opening/closing the mouth, pursing, raisingthe corner of the mouth (smiling), lowering the corner of the mouth,licking the lips, puckering and the like. Further, a deformation of afacial component may include any action, such as placing a finger on afacial component. Still further, a deformation of a facial component isnot specified to the foregoing.

Further, using an analysis result of the face features of the user andassociating with the interest degree for each facial component of thefacial image contained in the simulation image, interest degree sensor230 can more precisely estimate an interest of the user such as acomplex (feeling of inferiority).

Further, interest degree sensor 230 senses a satisfaction degree of theuser for a makeup simulation (a makeup image) being performed. Forexample, interest degree sensor 230 may sense a reaction of the userimmediately after a simulation image is displayed (after a lapse of orwithin a predetermined time, e.g., 10 seconds, at which a gut-reactionof the user is expected to appear, and the same will hold truehereinafter), to sense the satisfaction degree of the user in accordancewith such a sensing result. More specifically, interest degree sensor230 obtains the result in which the evaluation of the user (highlyvalued, poorly valued) is added to the interest degree. Interest degreesensor 230 records the sensed reactions of the user in chronologicalorder.

The user's reaction includes the angle and the deformation amount ofexpression parameters (direction of the eyes, face), involuntarygestures, voices or conversations, and living signals such as the bloodpressure or the heart rate of the user. For example, in the state wherethe seating position of the salesclerk is known, interest degree sensor230 senses whether or not eye contact is established between thesalesclerk and the user. Further, interest degree sensor 230 senses theangle of the user's face, to sense whether or not the user is lookingdown. Further, interest degree sensor 230 senses, as the user'sinvoluntary gesture, whether or not the user is thinking carefullyputting the user's hand on the face.

Interest degree information output unit 240 outputs, to second display260, interest degree information indicative of the interest degree(including the satisfaction degree) sensed by interest degree sensor230.

Based on the interest degree sensed by interest degree sensor 230,makeup candidate information output unit 250 outputs, to second display260, makeup candidate information indicative of the candidate for makeuprecommended to apply next. In the present exemplary embodiment, when theuser's interest degree for a makeup image contained in the simulationimage is high, makeup candidate information output unit 250 determines,as a candidate, makeup that corresponds to a facial componentcorresponding to the makeup image and that is different from makeupcorresponding to the makeup image. On the other hand, when the user'sinterest degree is low, makeup candidate information output unit 250determines, as a candidate, makeup for other facial component beingdifferent from the facial component corresponding to the makeup image.

Specifically, makeup candidate information output unit 250 determinesthe candidate for makeup recommended to apply next, according to: a keyperson, that is, the person who desired the process of generating thesimulation image (which is specified by the determination result bydetermination unit 300); a facial component sensed by interest degreesensor 230 to show a high interest degree (hereinafter also referred toas the “interested portion”); and a user's satisfaction degree for themakeup simulation being performed sensed by interest degree sensor 230.Makeup candidate information output unit 250 determines the candidatefor makeup recommended to apply next according to a makeup candidatedetermination table shown in FIG. 4.

FIG. 4 is the makeup candidate determination table according to thepresent exemplary embodiment. As shown in FIG. 4, when the key person isthe user, the interested portion is a makeup portion (a facial componentcorresponding to the makeup image), and the satisfaction degree is high,makeup candidate information output unit 250 determines also as acandidate, in addition to the makeup for the makeup portion, makeupwhich is different from the makeup. Note that, when the key person isthe user, the interested portion is a makeup portion, and thesatisfaction degree is high, makeup candidate information output unit250 desirably determines, as the makeup different from the makeup forthe makeup portion, makeup being similar in color to the makeup for themakeup portion, that is, makeup which is smaller in difference as acandidate as compared to the case when the key person is the salesclerk.

Further, when the key person is the user, the interested portion is amakeup portion, and the satisfaction degree is low, makeup candidateinformation output unit 250 determines as a candidate, in place of themakeup for the makeup portion, makeup different from the makeup.

Still further, when the key person is the user, the interested portionis a non-makeup portion (a facial component being different from thefacial component corresponding to the makeup image), and thesatisfaction degree is high, makeup candidate information output unit250 determines also as a candidate, in addition to the current makeupfor the makeup portion, makeup for a facial component corresponding tothe interested portion.

Still further, when the key person is the user, the interested portionis a non-makeup portion, and the satisfaction degree is low, makeupcandidate information output unit 250 determines as a candidate, inplace of the current makeup for the makeup portion, makeup for a facialcomponent corresponding to the interested portion.

Still further, when the key person is the salesclerk, the interestedportion is a makeup portion, and the satisfaction degree is high, makeupcandidate information output unit 250 determines also as a candidate, inaddition to the makeup for the makeup portion, makeup being differentfrom the makeup.

Still further, when the key person is the salesclerk, the interestedportion is a makeup portion, and a satisfaction degree is low, makeupcandidate information output unit 250 determines as a candidate, inplace of the makeup for the makeup portion, makeup being different fromthe makeup.

Still further, when the key person is the salesclerk, the interestedportion is a non-makeup portion, and the satisfaction degree is high,makeup candidate information output unit 250 determines as a candidate,in addition to the current makeup for the makeup portion, makeup for afacial component corresponding to the interested portion.

Still further, when the key person is the salesclerk, the interestedportion is a non-makeup portion, and the satisfaction degree is low,makeup candidate information output unit 250 determines as a candidate,in place of the current makeup for the makeup portion, makeup for afacial component corresponding to the interested portion.

Second display 260 is a liquid crystal display, an organic EL (organicelectroluminescence) display or the like. Second display 260 displays asalesclerk's window containing an interest degree information imagecorresponding to the interest degree information output from interestdegree information output unit 240, and a makeup candidate informationimage corresponding to the makeup candidate information output frommakeup candidate information output unit 250. FIG. 2B is a diagramshowing an exemplary display mode of salesclerk's window 500 containinginterest degree information image 520 and makeup candidate informationimage 540. As shown in FIG. 2B, salesclerk's window 500 contains, inaddition to interest degree information image 520 and makeup candidateinformation image 540, attribute information image 510 indicative of theuser's attribute information (name, age and the like), simulation image530 which is similar to simulation image 400 shown in FIG. 2A.Simulation image 400 and simulation image 530 are displayed while beingupdated in real time in synchronization with each other.

Interest degree information image 520 contains facial image 522 pickedup by image pickup unit 110, and heat map image 524 emphasizing a facialcomponent with which high interest degree is sensed by interest degreesensor 230. Further, interest degree information image 520 containsinformation (user's concern information such as thinner lips in theexample shown in FIG. 2B) relating to the characteristic of a facialcomponent (the lips in the example shown in FIG. 2B) corresponding tothe makeup image contained in simulation image 530. The informationrelating to the characteristic of a facial component is generated basedon the characteristic specified by facial component specifier 290.Further, interest degree information image 520 contains informationrelating to a change in the user's satisfaction degree for the makeupsimulation being performed, sensed by interest degree sensor 230 (3→2 inthe example shown in FIG. 2B). From the information contained ininterest degree information image 520, the salesclerk can easilyunderstand that the user's satisfaction degree for the makeup simulationbeing performed reduces, and the user's interested portion switches fromthe makeup portion (the lips) to the non-makeup portion (the eyes).

Makeup candidate information image 540 contains facial image 542 whichis picked up by image pickup unit 110 and in which a candidate formakeup recommended by makeup candidate information output unit 250 toapply next is displayed as being enhanced (makeup for the eyes in theexample shown in FIG. 2B). Further, in the present exemplary embodiment,makeup candidate information image 540 contains information indicativeof a predicted matching product which is predicted to match with themakeup recommended to apply next (predicted matching product ◯◯◯ in theexample shown in FIG. 2B). Note that, makeup candidate information image540 may contain information relating to makeup that can solve the user'sconcern indicated in the user's concern information contained ininterest degree information image 520. Further, makeup candidateinformation image 540 may contain stock information of the predictedmatching product (for example, in stock or out of stock, scheduleddelivery, whether or not being strategically promoted, and the like).From the information contained in makeup candidate information image540, the salesclerk can easily ascertain the candidate for makeuprecommended to apply next.

Second display 260 accepts operations of the salesclerk on salesclerk'sside terminal apparatus 200. To this end, second display 260 isstructured as a touchscreen monitor by a touchscreen being disposed tocover the window.

In the present exemplary embodiment, as means for accepting operationsof the salesclerk, in addition to the touchscreen monitor, a stylus (amanipulation unit) is separately provided. The stylus is provided with adiscrimination button for discriminating whether the process ofgenerating a simulation image by simulation image generator 210 isdesired by the user or the salesclerk (guiding the user). For example,when the process of generating a simulation image by simulation imagegenerator 210 is desired by the user, the salesclerk presses thediscrimination button provided at the stylus.

Facial component positional information storage 270 stores facialcomponent positional information indicative of the position and the likeof facial components in the average face.

Facial position sensor 280 senses the position of facial components inthe user's face based on a facial image picked up by image pickup unit110 of customer's side terminal apparatus 100.

Based on the facial component positional information stored in facialcomponent positional information storage 270 and the sensing result fromfacial position sensor 280, facial component specifier 290 specifies afacial component which is the characteristic of the user's face. Forexample, facial component specifier 290 may execute the schemeincluding: converting a facial component into a numerical value by theposition in the face, the positional relationship with other facialcomponent, the shape, the size, the color, and the like; comparing theobtained value against an average value of facial components in acertain number or greater similarly converted into values; andspecifying a facial component with which a difference exceeding apredetermined threshold value is detected as a facial component with acharacteristic. However, the present disclosure is not limited thereto.Makeup candidate information output unit 250 outputs, to second display260, makeup candidate information indicative of makeup for the facialcomponent specified by facial component specifier 290 as a candidate formakeup recommended to apply next.

Determination unit 300 determines whether or not the process ofgenerating the simulation image performed by simulation image generator210 is desired by the user, in accordance with whether or not thediscrimination button provided at the stylus is pressed. Note that,determination unit 300 may determine whether or not the process ofgenerating the simulation image performed by simulation image generator210 is desired by the user based on the analysis result on theconversation between the user and the salesclerk.

When determination unit 300 determines that the process of generatingthe simulation image is not desired by the user (that is, guided by thesalesclerk), interest degree history information recorder 310 stores, ininterest degree history information storage 320, interest degree historyinformation in which the facial component corresponding to the makeupimage contained in the simulation image and the interest degree sensedby interest degree sensor 230 in the current makeup simulation areassociated with each other.

FIG. 5A is a table showing the interest degree history informationaccording to the present exemplary embodiment. As shown in FIG. 5A,interest degree history information includes, as items, the date ofvisit, the makeup part of a facial component corresponding to a makeupimage, the color of the makeup part, the user's satisfaction degree formakeup corresponding to the makeup image, and the final purchase resultfor cosmetics corresponding to the makeup part. In the present exemplaryembodiment, as the value in the item “satisfaction degree”, one integervalue out of 1 to 5 is set. Further, as the value in the item “purchaseresult”, the mark indicative of purchase “◯”, or the mark indicative ofnon-purchase “x” is set. Note that, as shown in FIG. 5B, interest degreehistory information recorder 310 may store, irrespective of whether ornot the process of generating the simulation image is desired by theuser which is determined by determination unit 300, the interest degreehistory information of the current makeup simulation in interest degreehistory information storage 320.

As shown in FIGS. 5A and 5B, from the interest degree historyinformation relating to the key person “salesclerk”, since thesatisfaction degree is higher with blusher than other makeup part“lipstick”, it can be estimated that the user is following the guidanceof the salesclerk. Further, as shown in FIG. 5B, from the interestdegree history information relating to the makeup part “lipstick”, sincethe satisfaction degree is higher with the key person “user” than otherkey person “salesclerk”, it can be estimated that the user is inflexibleabout lipstick, and particularly shows strong preference for pink.Further, as shown in FIG. 5B, from the interest degree historyinformation relating to the makeup part “lipstick”, the key person“user”, the satisfaction degree “5”, and the purchase result “x”, it canbe estimated that, on the next visit of the user, it would be productiveto recommend “pink” color when lipstick becomes a conversation piece. Ashas been described above, by analyzing the interest degree historyinformation, the user's flexibility about any makeup part (whether theuser willingly follows the guidance of the salesclerk or becomesobstinate and inflexible) can be learned. Further, the user'sflexibility about the color of any makeup part can be learned. Notethat, it is also possible to analyze the probability of the userfollowing the makeup guided by the salesclerk for each makeup part andeach color (the user's flexibility), and the analysis result may bedisplayed on second display 260. Further, in the case of non-purchase ofthe makeup part on the previous visit of the user, this fact may bedisplayed on second display 260 together with the analysis result.

FIG. 6 is a flowchart showing an operation of salesclerk's side terminalapparatus 200 according to the present exemplary embodiment(corresponding to the “makeup simulation assistance method” of thepresent disclosure). The process shown in FIG. 6 starts in response toan execution request for a makeup simulation based on a manipulation ofthe user or the salesclerk on customer's side terminal apparatus 100.

Firstly, simulation image generator 210 acquires a facial image outputfrom image pickup unit 110 of customer's side terminal apparatus 100(step S100). Next, simulation image generator 210 generates a simulationimage obtained by superimposing, on the acquired facial image, a makeupimage representing the state of makeup applied to a facial component ofthe face contained in the acquired facial image (step S120).

Next, simulation image output unit 220 outputs the simulation imagegenerated by simulation image generator 210 to first display 120 ofcustomer's side terminal apparatus 100 (step S140). First display 120displays the simulation image output from customer's side terminalapparatus 100.

Next, interest degree sensor 230 senses, based on a facial image pickedup by image pickup unit 110 when the simulation image is output fromsimulation image output unit 220, the user's interest degree for themakeup image contained in the simulation image (step S160).

Next, interest degree information output unit 240 outputs, to seconddisplay 260, interest degree information indicative of the interestdegree sensed by interest degree sensor 230 (step S180). Next, based onthe interest degree sensed by interest degree sensor 230, makeupcandidate information output unit 250 outputs, to second display 260,makeup candidate information indicative of a candidate for makeuprecommended to apply next (step S200). Second display 260 displays theinterest degree information output from interest degree informationoutput unit 240, and the makeup candidate information output from makeupcandidate information output unit 250.

Finally, salesclerk's side terminal apparatus 200 determines whether ornot an end request for the makeup simulation has occurred based on amanipulation of the user or the salesclerk (step S220). As a result ofthe determination, when the end request for the makeup simulation hasnot occurred (step S220, NO), the process returns so as to precede stepS100. On the other hand, when the end request for the makeup simulationhas occurred (step S220, YES), salesclerk's side terminal apparatus 200ends the process in FIG. 6.

As has been described in detail above, salesclerk's side terminalapparatus 200 (the makeup simulation assistance apparatus) according tothe present exemplary embodiment includes: simulation image generator210 that generates a simulation image obtained by superimposing, on afacial image picked up by image pickup unit 110 that picks up an imageof a face of a user as the facial image, a makeup image showing a stateof makeup as being applied to a facial component of the face; asimulation image output unit 220 that outputs the simulation image beinggenerated; interest degree sensor 230 that senses an interest degree ofthe user for the makeup image, based on a facial image picked up byimage pickup unit 110 when the simulation image is output; and interestdegree information output unit 240 that outputs interest degreeinformation indicative of the interest degree being sensed.

With the present exemplary embodiment configured as above, when asimulation image is displayed, customer's side terminal apparatus 100identifies latent needs of the user (customer) for makeup, for example,hidden honest feeling or complex (feeling of inferiority), and providesthe salesclerk with information (interest degree information) which ishelpful in providing information that the user truly needs. Thus, thepresent exemplary embodiment can assist in face-to-face selling, toefficiently lead the customer to purchase of products relating to makeup(cosmetics).

Note that, in the above-described exemplary embodiment, while thedescription has been given of the case where customer's side terminalapparatus 100 and salesclerk's side terminal apparatus 200 areseparately embodied, they may be integrally embodied as one apparatus.In this case, one display may serve as both first display 120 and seconddisplay 260, and the CPUs as the processors may be one in number. Whentwo CPUs are used, functions such as generating and displaying asimulation image and an analysis such as sensing an interest degree maybe separately allotted to the CPUs.

Further, in the above-described exemplary embodiment, customer's sideterminal apparatus 100 may include, out of the elements of salesclerk'sside terminal apparatus 200, the elements other than second display 260.In this case, customer's side terminal apparatus 100 corresponds to the“makeup simulation assistance apparatus” of the present disclosure.

Still further, in above-described exemplary embodiment, makeup candidateinformation image 540 may include, based on the interest degree historyinformation and the like stored in interest degree history informationstorage 320, information relating to a candidate for makeup suitable forthe characteristic of the user's face (for example, enhancement for theeyes), or information indicative of past user's inflexibility (forexample, strong interest in color ◯◯).

Still further, in the above-described exemplary embodiment, whenface-to-face selling is started, the analysis result of the useraccording to the interest degree history information stored in interestdegree history information storage 320 may be displayed on seconddisplay 260. FIG. 7A shows an exemplary display mode of user's window600 at the start of face-to-face selling. As shown in FIG. 7A, theuser's window displays, for example, a general advertisement relating tocosmetics, or in consideration of the user, information relating tocosmetics that may attract the user.

FIG. 7B shows an exemplary display mode of salesclerk's window 700 atthe start of face-to-face selling. As shown in FIG. 7B, the salesclerk'swindow displays attribute information image 710 indicative of the user'sattribute information (name, age, previously visited shop, the attendingsalesclerk, and the like), previous simulation result image 720 showingthe result of the previous makeup simulation, and previous interestinformation image 730 showing the user's interest in the previous makeupsimulation.

The attribute information displayed on attribute information image 710is desirably information retrieved using the facial image picked up byimage pickup unit 110. At the start of face-to-face selling, thesalesclerk meticulously asking the user about the customer's informationfrom the beginning (such as the name and the age) may be offensive tothe user.

Previous simulation result image 720 contains user's facial image 724with makeup with a makeup part which shows non-purchase as the purchaseresult in the previous makeup simulation (lipstick in the example shownin FIG. 7B), and user's facial image 722 with makeup with a makeup partwhich shows purchase as the purchase result (blusher in the exampleshown in FIG. 7B). Facial image 722 contains enhanced image 726 whichenhances makeup with a makeup part. Further, previous simulation resultimage 720 also contains information relating to a non-purchase reason(great inflexibility as to color for lipstick, in the example shown inFIG. 7B), information showing a specific product name of the purchasedproduct (0 in the example shown in FIG. 7B).

Previous interest information image 730 contains user's facial image 732containing enhanced image 734 which enhances a facial component withwhich the user's interest degree has been high, and informationindicative of the fact that the interest degree for the facial componenthas been high (interested around the eyes, in the example shown in FIG.7B).

At the start of the face-to-face selling, by glancing over salesclerk'swindow 700, the salesclerk can note cautions in the current makeupsimulation before actually starting the makeup simulation, toefficiently lead the customer to purchase of cosmetics. In particular,also in the case where the attending salesclerk in a makeup simulationis switched between the previous simulation and the current simulationfrom, for example, an experienced salesclerk to an inexperiencedsalesclerk, a makeup simulation reflecting the previous makeupsimulation result can be performed.

Further, in the above-described exemplary embodiment, the correlationbetween the user's reaction immediately after a simulation image isdisplayed and the purchase result may be learned for each user, and theanalysis result according to the learning result may be displayed onsecond display 260. The correlation between the user's reaction and thepurchase result may be learned for each salesclerk.

Specifically, interest degree sensor 230 senses and records the user'sreaction immediately after a simulation image is displayed. Then, whenthere exist a plurality of purchase occasions during a past visit of theuser, interest degree sensor 230 detects the difference between theuser's reactions between a makeup simulation execution mode relating tocosmetics resulting in purchase and a makeup simulation execution moderelating to cosmetics not resulting in purchase, and extracts a user'sreaction exhibiting high correlation with purchase as a high evaluationsensing standard. When the extracted high evaluation sensing standard isalso sensed in a makeup simulation currently performed, interest degreesensor 230 notifies the makeup candidate information output unit 250 ofthe same. Then, the user's reaction corresponding to the high evaluationsensing standard (a facial component deformation), that is, the portionto be observed by the salesclerk, is displayed on second display 260(more specifically, with a dotted frame on facial image 522 contained ininterest degree information image 520). Note that, the high evaluationsensing standard may be displayed on the salesclerk's window as it is.Further, a user's reaction exhibiting high correlation with purchase maybe determined by, for example, a lapse of time since displaying asimulation image, such as a user's eye contact after a lapse of whatseconds results in purchase. Further, a user's reaction exhibiting highcorrelation with purchase may be correlated with a plurality of usershaving similar facial characteristics.

In the above-described exemplary embodiment, a makeup simulation usingcustomer's side terminal apparatus 100 may be performed at a retailcosmetics shop where face-to-face selling is performed, at a retailcosmetics shop where the customer determines purchase on his/her own, orat the home of the user. FIG. 8 shows a system configuration of thelatter two cases (no-salesclerk-selling). Being different from FIG. 1,no salesclerk is present, and therefore no second display (salesclerk'swindow) is provided. The makeup simulation to be performed next isdetermined by makeup candidate information output unit 250 withoutseeking salesclerk's final judgement, and the determination result istransmitted to simulation image generator 210. Desirably, the content ofthe makeup simulation is updated reflecting the sensing status of thecustomer's interest degree, for example at an interval of about 15seconds. On the other hand, when there exists any makeup simulation thatthe customer is interested in, the updating may be temporarily suspendedby any customer's manipulation such as touching. Further, the customermay change the content of the simulation such as color. Determinationunit 300 determines whether such a content of the makeup simulationperformed next is determined by system 10 according to the presentexemplary embodiment or by the customer. Similarly to the discriminatingwhether the “user” or the “salesclerk” is the key person in face-to-faceselling shown in FIGS. 5 and 6, this information is used indiscriminating whether the key person is the “user” or not, and asdescribed above, in determining the preference of the customer or theuser's flexibility.

Further, in this case, considering that the user's reaction in a makeupsimulation is assumed to vary depending on whether a makeup simulationusing customer's side terminal apparatus 100 is performed at a retailcosmetics shop or at the home of the user, the method of sensing theuser's interest degree by interest degree sensor 230 may be changed.Further, the method of sensing the user's interest degree by interestdegree sensor 230 may be changed depending on whether or not a makeupsimulation is performed in a time slot (for example, at nighttime,before going to bed) in which the user's latent needs (honest feeling)tend to appear as a facial expression.

Still further, in the above-described exemplary embodiment, not only theuser's face but also the whole body can be displayed on first display120. When any clothes or hairstyle are positioned in the direction ofthe line of sight of the user, the information indicative of the samemay be contained in interest degree information image 520. Further, inthe above-described exemplary embodiment, while the target of asimulation is just makeup in which makeup is virtually realized, thetarget of a simulation may be virtual coordination including a hairstyleand an outfit. When the target of a simulation is wider, it becomes moreimportant to ascertain the interest and reactions of the customer anddetermine the next action.

Still further, in the above-described exemplary embodiment, a makeupsimulation may be measured for each facial component, and thecorrelation with the purchase result for each facial component (forexample, a purchase failed to be made despite an extended performance ofa makeup simulation) may be recorded. Further, by extracting the timeslot which shows correlation with the purchase result, information thatmakes it possible to search for the salesclerk to whom the user caneasily (in short time) express his/her honest feeling may be recorded.

Still further, a plurality of facial images may be picked up for eachmakeup simulation (for example, an image of a smiling face, a face of abeauty, and the like), so that the salesclerk can select the facialimage to be contained in a purchase candidate presenting window which isdisplayed after a makeup simulation.

Still further, in the above-described exemplary embodiment, while thedescription has been given of an exemplary case where makeup simulationsystem 10 is used for identifying the user's interest (latent needs) formakeup, the present disclosure is not limited thereto. For example,makeup simulation system 10 may be used for analyzing cosmetics used inmakeup. More specifically, makeup simulation system 10 may be used foranalyzing whether a cosmetic item is expected to be purchased by anyuser, whether a cosmetic item has been purchased beyond all expectationsirrespective of the characteristics of the face, the user's flexibility,and the like upon recommendation of a salesclerk, and the like. Further,makeup simulation system 10 may be used for analyzing a salesclerk. Morespecifically, makeup simulation system 10 may be used for analyzingwhether or not the salesclerk is skilled in attracting interest of theuser through a makeup simulation, whether or not the communication skillof the salesclerk beaten low interest of the user and led to purchase,compensating for the low evaluation in the makeup simulation mode, andthe like.

Still further, the above-described exemplary embodiment is merely anexemplary embodiment in practicing the present disclosure, and thetechnical range of the present disclosure should not be construed aslimiting thereby. That is, the present disclosure can be practiced inany of various modes without departing from its spirit or its principlecharacteristics.

A makeup simulation assistance apparatus of the present disclosureincludes:

a simulation image generator that generates a simulation image obtainedby superimposing, on a facial image picked up by an image pickup unitthat picks up an image of a face of a user as the facial image, a makeupimage showing a state of makeup as being applied to a facial componentof the face;

a simulation image output unit that outputs the simulation image beinggenerated;

an interest degree sensor that senses an interest degree of the user forthe makeup image, based on a facial image picked up when the simulationimage is output; and

an interest degree information output unit that outputs interest degreeinformation indicative of the interest degree being sensed.

In the makeup simulation assistance apparatus, the interest degreesensor may sense the interest degree of the user by sensing a matchdegree between a facial component in which the user is interested and afacial component corresponding to the makeup image, based on the facialimage picked up when the simulation image is output.

In the makeup simulation assistance apparatus, the interest degree mayinclude a satisfaction degree of the user for the makeup image.

The makeup simulation assistance apparatus may further include a makeupcandidate information output unit that outputs makeup candidateinformation indicative of a candidate for makeup recommended to applynext, based on the interest degree being sensed.

In the makeup simulation assistance apparatus, the makeup candidateinformation output unit may output, when the interest degree is high,makeup candidate information indicative of makeup for the facialcomponent and being different from makeup corresponding to the makeupimage as the candidate, and the makeup candidate information output unitoutputs, when the interest degree is low, makeup candidate informationindicative of makeup for a facial component different from the facialcomponent as the candidate.

The makeup simulation assistance apparatus may further include:

a facial component positional information storage that stores facialcomponent positional information indicative of a position of a facialcomponent in an average face;

a facial position sensor that senses a position of a facial component inthe face of the user based on the facial image being picked up; and

a facial component specifier that specifies a facial component being acharacteristic of the face of the user based on the facial componentpositional information and a sensing result of the facial positionsensor,

wherein the makeup candidate information output unit may output makeupcandidate information indicative of makeup for the facial componentbeing specified as the candidate.

In the makeup simulation assistance apparatus, the interest degreeinformation output unit may output the interest degree information to asecond display that is different from a first display to which thesimulation image is output.

In the makeup simulation assistance apparatus, the interest degreesensor may sense a facial component of the face which is set to opposeto the image pickup unit by the user, and the interest degree may besensed based on a sensing result of the sensing.

In the makeup simulation assistance apparatus, the interest degreesensor may sense, out of facial components of the face displayed in thesimulation image, a facial component positioned in a direction of a lineof sight of the user, and may sense the interest degree based on asensing result of the sensing.

The makeup simulation assistance apparatus may further include:

a determination unit that determines whether or not the generating thesimulation image by the simulation image generator is desired by theuser; and

an interest degree history information recorder that stores, in aninterest degree history information storage when the determination unitdetermines that the generating is not desired by the user, interestdegree history information in which a facial component corresponding tothe makeup image and the interest degree being sensed are associatedwith each other.

In the makeup simulation assistance apparatus,

the determination unit may determine a user's flexibility being aprobability of the user following guidance of other person, based on theinterest degree history information and a purchase result, and

the interest degree information output unit may output a determinationresult relating to the user's flexibility.

In the makeup simulation assistance apparatus, the interest degreesensor may extract information relating to deformation of a facialcomponent based on the facial image picked up, acquire correlationbetween the extracted information relating to deformation of the facialcomponent after a lapse of a predetermined time since the simulationimage is output from the simulation image output unit and a purchaseresult, and may extract, based on the acquired correlation with apurchase result, deformation of the facial component showing correlationof at least a predetermined value with purchase and non-purchase as ahigh evaluation sensing standard and a low evaluation sensing standard,respectively.

A makeup simulation assistance method of the present disclosureincludes: generating a simulation image obtained by superimposing, on afacial image picked up by an image pickup unit that picks up an image ofa face of a user as the facial image, a makeup image showing a state ofmakeup as being applied to a facial component of the face;

outputting the simulation image being generated;

sensing an interest degree of the user for the makeup image, based on afacial image picked up when the simulation image is output; and

outputting interest degree information indicative of the interest degreebeing sensed.

The present disclosure provides a non-transitory computer-readablerecording medium storing a makeup simulation assistance program causinga computer to execute the steps of:

generating a simulation image obtained by superimposing, on a facialimage picked up by an image pickup unit that picks up an image of a faceof a user as the facial image, a makeup image showing a state of makeupas being applied to a facial component of the face;

outputting the simulation image being generated;

sensing an interest degree of the user for the makeup image, based onthe facial image picked up when the simulation image is output; and

outputting interest degree information indicative of the interest degreebeing sensed.

The present disclosure is useful as a makeup simulation assistanceapparatus, a makeup simulation assistance method, and a non-transitorycomputer-readable recording medium storing a makeup simulationassistance program each assisting in face-to-face selling andno-salesclerk-selling in order to efficiently derive purchase of aproduct relating to makeup.

What is claimed is:
 1. A makeup simulation assistance apparatuscomprising: a simulation image generator that generates a simulationimage obtained by superimposing, on a facial image picked up by an imagepickup unit that picks up an image of a face of a user as the facialimage, a makeup image showing a state of makeup as being applied to afacial component of the face; a simulation image output unit thatoutputs the simulation image being generated; an interest degree sensorthat senses an interest degree of the user for the makeup image, basedon a facial image picked up when the simulation image is output; aninterest degree information output unit that outputs interest degreeinformation indicative of the interest degree being sensed; adetermination unit that determines whether or not the generating thesimulation image by the simulation image generator is desired by theuser; and an interest degree history information recorder that stores,in an interest degree history information storage when the determinationunit determines that the generating is not desired by the user, interestdegree history information in which a facial component corresponding tothe makeup image and the interest degree being sensed are associatedwith each other.
 2. The makeup simulation assistance apparatus accordingto claim 1, wherein the interest degree sensor senses the interestdegree of the user by sensing a match degree between a facial componentin which the user is interested and a facial component corresponding tothe makeup image, based on the facial image picked up when thesimulation image is output.
 3. The makeup simulation assistanceapparatus according to claim 1, wherein the interest degree includes asatisfaction degree of the user for the makeup image.
 4. The makeupsimulation assistance apparatus according to claim 1, further comprisinga makeup candidate information output unit that outputs makeup candidateinformation indicative of a candidate for makeup recommended to applynext, based on the interest degree being sensed.
 5. The makeupsimulation assistance apparatus according to claim 4, wherein the makeupcandidate information output unit outputs, when the interest degree ishigh, makeup candidate information indicative of makeup for the facialcomponent and being different from makeup corresponding to the makeupimage as the candidate, and the makeup candidate information output unitoutputs, when the interest degree is low, makeup candidate informationindicative of makeup for a facial component different from the facialcomponent as the candidate.
 6. The makeup simulation assistanceapparatus according to claim 4, further comprising: a facial componentpositional information storage that stores facial component positionalinformation indicative of a position of a facial component in an averageface; a facial position sensor that senses a position of a facialcomponent in the face of the user based on the facial image being pickedup; and a facial component specifier that specifies a facial componentbeing a characteristic of the face of the user based on the facialcomponent positional information and a sensing result of the facialposition sensor, wherein the makeup candidate information output unitoutputs makeup candidate information indicative of makeup for the facialcomponent being specified as the candidate.
 7. The makeup simulationassistance apparatus according to claim 1, wherein the interest degreeinformation output unit outputs the interest degree information to asecond display that is different from a first display to which thesimulation image is output.
 8. The makeup simulation assistanceapparatus according to claim 1, wherein the interest degree sensorsenses a facial component of the face which is set to oppose to theimage pickup unit by the user, and the interest degree is sensed basedon a sensing result of the sensing.
 9. The makeup simulation assistanceapparatus according to claim 1, wherein the interest degree sensorsenses, out of facial components of the face displayed in the simulationimage, a facial component positioned in a direction of a line of sightof the user, and senses the interest degree based on a sensing result ofthe sensing.
 10. The makeup simulation assistance apparatus according toclaim 1, wherein the determination unit determines a user's flexibilitybeing a probability of the user following guidance of other person,based on the interest degree history information and a purchase result,and the interest degree information output unit outputs a determinationresult relating to the user's flexibility.
 11. The makeup simulationassistance apparatus according to claim 1, wherein the interest degreesensor extracts information relating to deformation of a facialcomponent based on the facial image picked up, acquires correlationbetween the extracted information relating to deformation of the facialcomponent after a lapse of a predetermined time since the simulationimage is output from the simulation image output unit and a purchaseresult, and extracts, based on the acquired correlation with a purchaseresult, deformation of the facial component showing correlation of atleast a predetermined value with purchase and non-purchase as a highevaluation sensing standard and a low evaluation sensing standard,respectively.
 12. A makeup simulation assistance method comprising:generating a simulation image obtained by superimposing, on a facialimage picked up by an image pickup unit that picks up an image of a faceof a user as the facial image, a makeup image showing a state of makeupas being applied to a facial component of the face; outputting thesimulation image being generated; sensing an interest degree of the userfor the makeup image, based on a facial image picked up when thesimulation image is output; outputting interest degree informationindicative of the interest degree being sensed; determining whether ornot the generating the simulation image is desired by the user; andstoring, in an interest degree history information storage when thegenerating is not desired by the user, interest degree historyinformation in which a facial component corresponding to the makeupimage and the interest degree being sensed are associated with eachother.
 13. A non-transitory computer-readable recording medium storing amakeup simulation assistance program causing a computer to execute thesteps of: generating a simulation image obtained by superimposing, on afacial image picked up by an image pickup unit that picks up an image ofa face of a user as the facial image, a makeup image showing a state ofmakeup as being applied to a facial component of the face; outputtingthe simulation image being generated; sensing an interest degree of theuser for the makeup image, based on the facial image picked up when thesimulation image is output; outputting interest degree informationindicative of the interest degree being sensed; determining whether ornot the generating the simulation image is desired by the user; andstoring, in an interest degree history information storage when thegenerating is not desired by the user, interest degree historyinformation in which a facial component corresponding to the makeupimage and the interest degree being sensed are associated with eachother.